Data loss recovery in a secondary storage controller from a primary storage controller

ABSTRACT

A secondary storage controller determines one or more tracks of one or more volumes in which data loss has occurred in the secondary storage controller. The secondary storage controller suspends a peer to peer remote copy operation between the secondary storage controller and a primary storage controller. Information on the one or more tracks of the one or more volumes in which the data loss has occurred is transmitted to the primary storage controller.

BACKGROUND 1. Field

Embodiments relate to mechanisms for data loss recovery in a secondarystorage controller from a primary storage controller.

2. Background

A storage controller may control access to storage for one or more hostcomputational devices that may be coupled to the storage system over anetwork. A storage management application that executes in the storagecontroller may manage a plurality of storage devices, such as diskdrives, tape drives, flash drives, direct access storage devices (DASD),etc., that are coupled to the storage system. A host may sendInput/Output (I/O) commands to the storage controller and the storagecontroller may execute the I/O commands to read data from the storagedevices or write data to the storage devices.

Peer to Peer Remote Copy or PPRC is a protocol to replicate a storagevolume from one storage controller to another storage controller in aremote site. Synchronous PPRC causes each write to the primary volume tobe performed to the secondary volume as well, and the input/output (I/O)is only considered complete when update to both primary and secondaryhave completed. Asynchronous PPRC may flag tracks on the primary storagecontroller to be duplicated to the secondary storage controller whentime permits.

SUMMARY OF THE PREFERRED EMBODIMENTS

Provided are a method, a system, a computer program product in which asecondary storage controller determines one or more tracks of one ormore volumes in which data loss has occurred in the secondary storagecontroller. The secondary storage controller suspends a peer to peerremote copy operation between the secondary storage controller and aprimary storage controller. Information on the one or more tracks of theone or more volumes in which the data loss has occurred is transmittedto the primary storage controller.

In certain embodiments, the secondary storage controller accumulatesover a period of time the information on the one or more tracks of theone or more volumes in which the data loss has occurred, prior totransmitting of any part of the information to the primary storagecontroller.

In further embodiments, in response to an unsuspension of the peer topeer remote copy operation, the secondary storage controller receivesvia the peer to peer remote copy operation, data stored in the primarystorage controller to restore data that was lost in the one or moretracks of the one or more volumes of the secondary storage controller.

In additional embodiments, the one or more tracks comprise a first trackand a second track of a volume. In response to an unsuspension of thepeer to peer remote copy operation, the secondary storage controllerreceives data stored in the primary storage controller to restore datain a range of tracks starting from the first track to the second trackof the volume of the secondary storage controller to recover from thedata loss.

In further embodiments, existing code to perform the peer to peer remotecopy operation between the primary storage controller and the secondarystorage controller remains unchanged in the primary storage controller,wherein a data structure is updated in the primary storage controller toindicate that data of the one or more tracks of the one or more volumesare to be transmitted on continuation of the peer to peer remote copyoperation to the secondary storage controller.

In yet additional embodiments, a tertiary storage controller determinesa set of tracks of a volume in which data loss has occurred in thetertiary storage controller. The tertiary storage controller suspendsanother peer to peer remote copy operation between the tertiary storagecontroller and the secondary storage controller. The tertiary storagecontroller transmits information on the set of tracks of the volume inwhich the data loss has occurred to the secondary storage controller.

In further embodiments, the secondary storage controller is a firstsecondary storage controller, wherein a multi-target peer to peer remotecopy operation is in progress from the primary storage controller to thefirst secondary storage controller and a second secondary storagecontroller. The second secondary storage controller determines a set oftracks of a volume in which data loss has occurred in the secondsecondary storage controller. The second secondary storage controllersuspends another peer to peer remote copy operation between the secondsecondary storage controller and the primary storage controller. Thesecond secondary storage controller transmits information on the set oftracks of the volume in which data loss has occurred to the primarystorage controller.

BRIEF DESCRIPTION OF THE DRAWINGS

Referring now to the drawings in which like reference numbers representcorresponding parts throughout:

FIG. 1 illustrates a block diagram of a computing environment comprisinga primary storage controller and a secondary storage controller in apeer to peer remote copy relationship, in accordance with certainembodiments;

FIG. 2 illustrates a flowchart of operations performed by a secondarystorage controller in the computing environment of FIG. 1 to recoverfrom a data loss, in accordance with certain embodiments;

FIG. 3 illustrates a block diagram of a computing environment comprisinga primary storage controller, a secondary storage controller, and atertiary storage controller, in accordance with certain embodiments;

FIG. 4 illustrates a flowchart of operations performed by a tertiarystorage controller in the computing environment of FIG. 3 to recoverfrom a data loss in accordance with certain embodiments;

FIG. 5 illustrates a block diagram of a computing environment comprisinga primary storage controller in peer to peer remote copy relationshipswith a first secondary storage controller and a second secondary storagecontroller, in accordance with certain embodiments;

FIG. 6 illustrates a flowchart of operations performed by a the firststorage secondary controller and the second secondary storage controllerin the computing environment of FIG. 5 to recover from a data loss, inaccordance with certain embodiments;

FIG. 7 illustrates a block diagram of a cloud computing environment, inaccordance with certain embodiments;

FIG. 8 illustrates a block diagram of further details of the cloudcomputing environment of FIG. 7 , in accordance with certainembodiments; and

FIG. 9 illustrates a block diagram of a computational system that showscertain elements that may be included in the storage system and/or thehost(s), as described in FIGS. 1-8 , in accordance with certainembodiments.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanyingdrawings which form a part hereof and which illustrate severalembodiments. It is understood that other embodiments may be utilized andstructural and operational changes may be made.

In high end storage solutions one of the most important design points isto avoid data loss. Unfortunately even with technologies such asRedundant Array of Independent Disks (RAID), data loss can still occurunder certain situations. One such scenario is during an RAID arrayrebuild in which if any of the drives in the same array encounter anunrecoverable media error, then data loss may be encountered as the datacan no longer be reconstructed for that portion of the array.

In certain storage systems copy service functionality allows formultiple copies of the data to be maintained asynchronously orsynchronously. Thus in certain storage solutions that deploy copyservice technology, if data is lost due to media errors on one storagesystem, then another copy of the data exists on a remote storage system.However there is no mechanism to restore the data automatically, and thecustomer has to go through a manual process to restore the data from abackup copy. Certain embodiments provide mechanisms to recovery fromdata loss on a peer to peer remote copy (PPRC) secondary volume(s)automatically. Currently when data loss occurs on a PPRC secondaryvolume the entire volume has to be manually restored from backup. Ifmany volumes have been affected or if the volumes are very large thisprocess can take a very long time, and thus the recovery time may beextended. In certain embodiments, the storage systems keep track ofwhich tracks have been lost and only restores those tracks or a range oftracks for the given volume.

In certain embodiments, when data loss occurs on a PPRC secondaryvolume, mechanisms are provided to keep track of all the tracks thathave been lost. This list of tracks are used to create a range of tracksfor each affected volume that need to be restored from the primarystorage system. This range of tracks for each volume is transferred tothe primary storage system. On the primary storage system this list oftracks is added to a data structure, such as a data structure called theout of synchronization (OOS) bit map. When the OOS bit map is processedthe range of tracks that are needed to recover the data loss is sentfrom the primary system to the secondary system, thus recovering fromthe data loss on the secondary system.

Exemplary Embodiments

FIG. 1 illustrates a block diagram of a computing environment 100comprising a primary storage controller 102 coupled to a secondarystorage controller 104. The primary storage controller 102 and thesecondary storage controller 104 are configurable to be in communicationwith one more hosts communicates with a plurality of hosts 106 over anetwork, in accordance with certain embodiments.

The storage controllers 102, 104 and the hosts 106 may comprise anysuitable computational device including those presently known in theart, such as, a personal computer, a workstation, a server, a mainframe,a hand held computer, a palm top computer, a telephony device, a networkappliance, a blade computer, a processing device, a controller, etc.

The storage controllers 102, 104 and the hosts 106 may be elements inany suitable network, such as, a storage area network, a wide areanetwork, the Internet, an intranet. In certain embodiments, the storagecontrollers 102, 104 and the hosts 106 may be elements in a cloudcomputing environment.

The primary storage controller 102 may manage a plurality of storagedevices 108 and the secondary storage controller 104 may manage aplurality of storage device 110. The primary storage controller 102 mayinclude a peer to peer remote copy application 112 and the secondarystorage controller 104 may also include a peer to peer remote copyapplication 114. The peer to peer remote copy applications 112, 114allow peer to peer remote copy operations from the primary storagecontroller 102 to the secondary storage controller 104 (as shown viareference numeral 116).

The primary storage controller maintains a data structure 118 that maycomprise an OOS bitmap to synchronize primary storage controller volumesto secondary storage controller volumes during a peer to peer remotecopy operations.

In certain embodiments, the secondary storage controller 104 mayaccumulate information in a data structure 120 on one or more tracks ofone or more volumes in which data loss has occurred.

FIG. 2 illustrates a flowchart 200 of operations performed by thesecondary storage controller 104 in the computing environment 100 ofFIG. 1 to recover from a data loss, in accordance with certainembodiments.

Control starts at block 202 in which the secondary storage controller104 determines one or more tracks of one or more volumes in which dataloss has occurred in the secondary storage controller 104. The secondarystorage controller 104 suspends (at block 204) a peer to peer remotecopy operation 116 between the secondary storage controller 104 and theprimary storage controller 102.

Control proceeds to block 206, in which the secondary storage controller104 accumulates over a period of time the information on the one or moretracks of the one or more volumes in which the data loss has occurred.Control proceeds to block 208, in which the accumulated information 120on the one or more tracks of the one or more volumes in which the dataloss has occurred is transmitted to the primary storage controller 102.By accumulating the information over a period of time and transmittingthe accumulated information processing time is saved in the secondarystorage controller 104 and the primary storage controller 102, asrepeated interruptions of the primary storage controller 102 from thesecondary storage controller 104 is avoided.

From block 208, control proceeds to block 210, where an unsuspension ismade by the secondary storage controller 104 of the suspended peer topeer remote copy operation 116. In response the unsuspension of the peerto peer remote copy operation 116, the secondary storage controller 104receives via the peer to peer remote copy operation 116, data stored inthe primary storage controller 102 to restore data that was lost in theone or more tracks of the one or more volumes of the secondary storagecontroller 104.

To perform the operations shown in FIG. 2 , existing code to perform thepeer to peer remote copy operation between the primary storagecontroller 102 and the secondary storage controller 104 remainsunchanged in the primary storage controller 102. The data structure 118(e.g., an OOS bitmap) is updated in the primary storage controller 102on receiving the accumulated information 120 to indicate that data ofthe one or more tracks of the one or more volumes are to be transmittedon continuation of the peer to peer remote copy operation to thesecondary storage controller 104.

FIG. 3 illustrates a block diagram of a computing environment 300comprising a primary storage controller 302, a secondary storagecontroller 304, and a tertiary storage controller 304, in accordancewith certain embodiments. The primary storage controller 302 is in apeer to peer remote copy relationship 308 with the secondary storagecontroller 304, and the secondary storage controller 304 is in a peer topeer remote copy relationship 310 with the tertiary storage controller306.

Therefore, in FIG. 3 storage volumes are copied from the primary storagecontroller 302 to the secondary storage controller 304, and also fromthe secondary storage controller 304 to the tertiary storage controller306.

FIG. 4 illustrates a flowchart 400 of operations performed by a tertiarystorage controller 306 in the computing environment 300 of FIG. 3 torecover from a data loss in accordance with certain embodiments.

Control starts at block 402 in which the tertiary storage controller 306determines a set of tracks of a volume in which data loss has occurredin the tertiary storage controller 306. The tertiary storage controller306 suspends (at block 404) a peer to peer remote copy operation 310between the tertiary storage controller 306 and the secondary storagecontroller 304. The tertiary storage controller 306 then transmits (atblock 406) information on the set of tracks of the volume in which thedata loss has occurred to the secondary storage controller 304.

From block 406, control proceeds to block 408, where an unsuspension ismade by the tertiary storage controller 306 of the suspended peer topeer remote copy operation 310. In response the unsuspension of the peerto peer remote copy operation 310, the tertiary storage controller 104receives via the peer to peer remote copy operation 310, data stored inthe secondary storage controller 304 to restore data that was lost inthe one or more tracks of the one or more volumes of the tertiarystorage controller 306.

FIG. 5 illustrates a block diagram of a computing environment 500comprising a primary storage controller 502 in peer to peer remote copyrelationships 508, 510 with a first secondary storage controller 504 anda second secondary storage controller 506, in accordance with certainembodiments. Volumes are copied from the primary storage controller 502to the first secondary storage controller 504 via peer to peer remotecopy 508. Additionally volumes may also be copied from the primarystorage controller 502 to the second secondary storage controller 506via peer to peer remote copy 510.

FIG. 6 illustrates a flowchart 600 of operations performed by a thefirst secondary storage controller 504 and the second secondary storagecontroller 506 in the computing environment 500 of FIG. 5 to recoverfrom data loss, in accordance with certain embodiments.

Control starts at block 602 and proceeds in parallel to block 604 and606.

At block 604, the first secondary storage controller 504 determines oneor more tracks of one or more volumes in which data loss has occurred inthe first secondary storage controller 504. The first secondary storagecontroller 504 suspends (at block 608) a peer to peer remote copyoperation 508 between the first secondary storage controller 504 and theprimary storage controller 502.

From block 608 control proceeds to block 610, in which the firstsecondary storage controller 504 first accumulates over a period of timethe information on the one or more tracks of the one or more volumes inwhich the data loss has occurred, and then transmits the accumulatedinformation on the one or more tracks of the one or more volumes inwhich the data loss has occurred to the primary storage controller 502.By accumulating the information over a period of time and transmittingthe accumulated information processing time is saved in the firstsecondary storage controller 504 and the primary storage controller 502,as repeated interruptions of the primary storage controller 502 from thefirst secondary storage controller 504 is avoided.

From block 610, control proceeds to block 612 where an unsuspension ismade by the first secondary storage controller 504 of the suspended peerto peer remote copy operation 508. In response the unsuspension of thepeer to peer remote copy operation 508, the first secondary storagecontroller 504 receives via the peer to peer remote copy operation 508,data stored in the primary storage controller 502 to restore data thatwas lost in the one or more tracks of the one or more volumes of thefirst secondary storage controller 504. Then the process stops (at block614).

In parallel to the operations 604, 608, 610, 612 being performed by thefirst secondary storage controller 504, the second secondary storagecontroller 506 may perform operations shown in blocks 606, 616, 618,620.

At block 606 the second secondary storage controller 506 determines oneor more tracks of one or more volumes in which data loss has occurred inthe second secondary storage controller 506. The second secondarystorage controller 506 suspends (at block 616) a peer to peer remotecopy operation 510 between the second secondary storage controller 506and the primary storage controller 502.

From block 616 control proceeds to block 618, in which the secondsecondary storage controller 506 accumulates over a period of time theinformation on the one or more tracks of the one or more volumes inwhich the data loss has occurred. In block 618 the accumulatedinformation on the one or more tracks of the one or more volumes inwhich the data loss has occurred is transmitted to the primary storagecontroller 502. By accumulating the information over a period of timeand transmitting the accumulated information processing time is saved inthe second secondary storage controller 506 and the primary storagecontroller 502, as repeated interruptions of the primary storagecontroller 502 from the second secondary storage controller 506 isavoided.

From block 618, control proceeds to block 620 where an unsuspension ismade by the second secondary storage controller 506 of the suspendedpeer to peer remote copy operation 510. In response the unsuspension ofthe peer to peer remote copy operation 510, the second secondary storagecontroller 506 receives via the peer to peer remote copy operation 510,data stored in the primary storage controller 502 to restore data thatwas lost in the one or more tracks of the one or more volumes of thesecond secondary storage controller 506. Then the process stops (atblock 614).

To perform the operations shown in FIG. 6 , existing code to perform thepeer to peer remote copy operations 508, 510 between the primary storagecontroller 502 and the first and second secondary storage controllers504, 506 remains unchanged in the primary storage controller 502. Datastructures (e.g., an OOS bitmaps) are updated in the primary storagecontroller 502 on receiving the accumulated information from the firstsecondary storage controller 504 and/or the second secondary storagecontroller 506 to indicate that data of the one or more tracks of theone or more volumes are to be transmitted from the primary storagecontroller 502 on continuation of one or more of the peer to peer remotecopy operation 508, 510.

Therefore, FIGS. 1-6 illustrate embodiments to recovery from data lossin one or more secondary storage controllers and in tertiary storagecontrollers.

Additional Embodiments

In certain embodiments, if data loss happens on enterprise storagesystem the following is performed by an application. The applicationchecks to see if a lost track is a PPRC secondary track and if it is,then the application performs the following:

(1) Create a list of each PPRC secondary track that has been affected.(2) Create a list that keeps track of all the volumes that encountereddata loss, and a range of track that needs to be recovered for thatvolume from the primary system (a startTrack and endTrack datastructures show the range).(3) Send a message to the host system for each affected volume to notifythe host that data loss has occurred and a remote copy restore will beattempted.(4) Queue a timer to process the lost track volume list for a givenamount of time (e.g. 5 seconds). For each additional track that is lostthis timer is requeued for another 5 seconds. Thus the task to processthe list will get dispatched 5 seconds after the last data loss trackhas been seen, or at least no more data loss has been seen for 5seconds.(5) When the “process volume list” task is dispatched (once no data losshas been seen for 5 seconds) do the following: (a) Cycle through thelist and for each affected volume; (b) Call the copy service codecomponent with a range of tracks (startTrack and endTrack that need tobe restored from the remote copy of data; (c) The copy service componentwill add these volumes an range of tracks to their meta data “VolumesNeeding Restored” structure. This structure is non-volatile so once itis in this metadata even if the system reboots or shuts down the losttracks will be restored at some point; (d) The copy service componentthen suspends the PPRC relationship. The suspend reason code indicatesthat this suspension was done because data loss occurred on thesecondary system and on un-suspend of the PPRC relationship the datawill be recovered; (e) the copy service code component then calls backthe calling component to indicate that this volume and range of trackshas been successfully added to the copy service meta data structure; (f)the copy service code updates the volume list log for this volume withthe status that it has been added to the copy service meta datastructure.(6) If more tracks encounter data loss at this point the process theprevious process repeats.(7) 10 Minutes after there has been no data loss process the track logfile. This file keeps track of every single track that has been lost andwhether or not it will be restored.(8) Using the volume list log cycle through the track log and update itindicating which tracks will be restored from the remote copy of thedata (the primary copy). Write this file out to the file system so ifneeded it can be analyzed at a later time (for debug purposes).(9) Start a task that queries the state of each volume that was affectedwith data loss. This task will check every so often to determine whenthe data has been restored (when the secondary copy is insynchronization with the primary copy).

In certain embodiments, when the PPRC relationship un-suspends theapplications performs the following:

(1) Cycle through the copy service Volumes Needing Restored structureand for each volume do the following: (a) Send a message to the remotesystem (primary system) indicating the volume number and range of tracksthat need to be restored; (b) Upon receiving this message the remotesystem (primary system) updates its OOS bit map for this volume with therange of tracks that need restored; (c) On the next primaryresynchronization of the data all tracks that are in the OOS for thisvolume are sent to the secondary system.(2) The task that is querying the state of each affected volume detectsthat the local copy is now in synchronization with the primary copy andthus the data has now been recovered and sends a notification to thehost that all data has now been restored.

Cloud Computing Environment

Cloud computing is a model for enabling convenient, on-demand networkaccess to a shared pool of configurable computing resources (e.g.,networks, servers, storage, applications, and services) that can berapidly provisioned and released with minimal management effort orservice provider interaction.

Referring now to FIG. 7 , an illustrative cloud computing environment 50is depicted. As shown, cloud computing environment 50 comprises one ormore cloud computing nodes 10 with which local computing devices used bycloud consumers, such as, for example, personal digital assistant (PDA)or cellular telephone 54A, desktop computer 54B, laptop computer 54C,and/or automobile computer system 54N may communicate. Nodes 10 maycommunicate with one another. They may be grouped (not shown) physicallyor virtually, in one or more networks, such as Private, Community,Public, or Hybrid clouds as described hereinabove, or a combinationthereof. This allows cloud computing environment 50 to offerinfrastructure, platforms and/or software as services for which a cloudconsumer does not need to maintain resources on a local computingdevice. It is understood that the types of computing devices 54A-N shownin FIG. 7 are intended to be illustrative only and that computing nodes10 and cloud computing environment 50 can communicate with any type ofcomputerized device over any type of network and/or network addressableconnection (e.g., using a web browser).

Referring now to FIG. 8 , a set of functional abstraction layersprovided by cloud computing environment 50 (FIG. 7 ) is shown. It shouldbe understood in advance that the components, layers, and functionsshown in FIG. 8 are intended to be illustrative only and embodiments ofthe invention are not limited thereto.

Hardware and software layer 60 includes hardware and softwarecomponents. Examples of hardware components include mainframes, in oneexample IBM zSeries* systems; RISC (Reduced Instruction Set Computer)architecture based servers, in one example IBM pSeries* systems; IBMxSeries* systems; IBM BladeCenter* systems; storage devices; networksand networking components. Examples of software components includenetwork application server software, in one example IBM WebSphere*application server software; and database software, in one example IBMDB2* database software. * IBM, zSeries, pSeries, xSeries, BladeCenter,WebSphere, and DB2 are trademarks of International Business MachinesCorporation registered in many jurisdictions worldwide.

Virtualization layer 62 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers;virtual storage; virtual networks, including virtual private networks;virtual applications and operating systems; and virtual clients.

In one example, management layer 64 may provide the functions describedbelow. Resource provisioning provides dynamic procurement of computingresources and other resources that are utilized to perform tasks withinthe cloud computing environment. Metering and Pricing provide costtracking as resources are utilized within the cloud computingenvironment, and billing or invoicing for consumption of theseresources. In one example, these resources may comprise applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal provides access to the cloud computing environment forconsumers and system administrators. Service level management providescloud computing resource allocation and management such that requiredservice levels are met. Service Level Agreement (SLA) planning andfulfillment provide pre-arrangement for, and procurement of, cloudcomputing resources for which a future requirement is anticipated inaccordance with an SLA.

Workloads layer 66 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation; software development and lifecycle management; virtualclassroom education delivery; data analytics processing; transactionprocessing; and the driver software and peer to peer remote copyprocessing 68 as shown in FIGS. 1-7 .

Additional Embodiment Details

The described operations may be implemented as a method, apparatus orcomputer program product using standard programming and/or engineeringtechniques to produce software, firmware, hardware, or any combinationthereof. Accordingly, aspects of the embodiments may take the form of anentirely hardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,aspects of the embodiments may take the form of a computer programproduct. The computer program product may include a computer readablestorage medium (or media) having computer readable program instructionsthereon for causing a processor to carry out aspects of the presentembodiments.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present embodiments may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present embodiments.

Aspects of the present embodiments are described herein with referenceto flowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instruction.

FIG. 9 illustrates a block diagram that shows certain elements that maybe included in storage controllers shown or used in FIGS. 1-6 , or othercomputational devices in accordance with certain embodiments. The system900 may include a circuitry 902 that may in certain embodiments includeat least a processor 904. The system 900 may also include a memory 906(e.g., a volatile memory device), and storage 908. The storage 908 mayinclude a non-volatile memory device (e.g., EEPROM, ROM, PROM, flash,firmware, programmable logic, etc.), magnetic disk drive, optical diskdrive, tape drive, etc. The storage 908 may comprise an internal storagedevice, an attached storage device and/or a network accessible storagedevice. The system 900 may include a program logic 910 including code912 that may be loaded into the memory 906 and executed by the processor904 or circuitry 902. In certain embodiments, the program logic 910including code 912 may be stored in the storage 908. In certain otherembodiments, the program logic 910 may be implemented in the circuitry902. One or more of the components in the system 900 may communicate viaa bus or via other coupling or connection 914. Therefore, while FIG. 9shows the program logic 910 separately from the other elements, theprogram logic 910 may be implemented in the memory 906 and/or thecircuitry 902.

Certain embodiments may be directed to a method for deploying computinginstruction by a person or automated processing integratingcomputer-readable code into a computing system, wherein the code incombination with the computing system is enabled to perform theoperations of the described embodiments.

The terms “an embodiment”, “embodiment”, “embodiments”, “theembodiment”, “the embodiments”, “one or more embodiments”, “someembodiments”, and “one embodiment” mean “one or more (but not all)embodiments of the present invention(s)” unless expressly specifiedotherwise.

The terms “including”, “comprising”, “having” and variations thereofmean “including but not limited to”, unless expressly specifiedotherwise.

The enumerated listing of items does not imply that any or all of theitems are mutually exclusive, unless expressly specified otherwise.

The terms “a”, “an” and “the” mean “one or more”, unless expresslyspecified otherwise.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or moreintermediaries.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Onthe contrary a variety of optional components are described toillustrate the wide variety of possible embodiments of the presentinvention.

Further, although process steps, method steps, algorithms or the likemay be described in a sequential order, such processes, methods andalgorithms may be configured to work in alternate orders. In otherwords, any sequence or order of steps that may be described does notnecessarily indicate a requirement that the steps be performed in thatorder. The steps of processes described herein may be performed in anyorder practical. Further, some steps may be performed simultaneously.

When a single device or article is described herein, it will be readilyapparent that more than one device/article (whether or not theycooperate) may be used in place of a single device/article. Similarly,where more than one device or article is described herein (whether ornot they cooperate), it will be readily apparent that a singledevice/article may be used in place of the more than one device orarticle or a different number of devices/articles may be used instead ofthe shown number of devices or programs. The functionality and/or thefeatures of a device may be alternatively embodied by one or more otherdevices which are not explicitly described as having suchfunctionality/features. Thus, other embodiments of the present inventionneed not include the device itself.

At least certain operations that may have been illustrated in thefigures show certain events occurring in a certain order. In alternativeembodiments, certain operations may be performed in a different order,modified or removed. Moreover, steps may be added to the above describedlogic and still conform to the described embodiments. Further,operations described herein may occur sequentially or certain operationsmay be processed in parallel. Yet further, operations may be performedby a single processing unit or by distributed processing units.

The foregoing description of various embodiments of the invention hasbeen presented for the purposes of illustration and description. It isnot intended to be exhaustive or to limit the invention to the preciseform disclosed. Many modifications and variations are possible in lightof the above teaching. It is intended that the scope of the invention belimited not by this detailed description, but rather by the claimsappended hereto. The above specification, examples and data provide acomplete description of the manufacture and use of the composition ofthe invention. Since many embodiments of the invention can be madewithout departing from the spirit and scope of the invention, theinvention resides in the claims hereinafter appended.

1-20. (canceled)
 21. A method comprising, determining, by a secondarystorage controller, one or more tracks of one or more volumes in whichdata loss has occurred in the secondary storage controller; suspending,by the secondary storage controller, a peer to peer remote copyoperation between the secondary storage controller and a primary storagecontroller; and transmitting information on the one or more tracks ofthe one or more volumes in which the data loss has occurred to theprimary storage controller, wherein a timer is queued to process a losttrack volume list for a predetermined amount of time, and for eachadditional track that is lost the timer is requeued once again for thepredetermined amount of time to dispatch a task to process the losttrack volume list once no data loss has been observed for at least thepredetermined amount of time.
 22. The method of claim 21, wherein inresponse to a process volume list task being dispatched once no dataloss has been detected for the predetermined amount of time, cyclingthrough a list and for volumes affected by data loss calling a copyservice code component with a range of peer to peer remote copysecondary tracks.
 23. The method of claim 21, the method furthercomprising: accumulating over a period of time, by the secondary storagecontroller, the information on the one or more tracks of the one or morevolumes in which the data loss has occurred, prior to transmitting ofany part of the information to the primary storage controller.
 24. Themethod of claim 21, wherein the one or more tracks comprise a firsttrack and a second track of a volume, the method further comprising: inresponse to an unsuspension of the peer to peer remote copy operation,receiving by the secondary storage controller, data stored in theprimary storage controller to restore data in a range of tracks startingfrom the first track to the second track of the volume of the secondarystorage controller to recover from the data loss.
 25. The method ofclaim 21, wherein a data structure is updated in the primary storagecontroller to indicate that data of the one or more tracks of the one ormore volumes are to be transmitted on continuation of the peer to peerremote copy operation to the secondary storage controller.
 26. Themethod of claim 21, the method further comprising: determining, by atertiary storage controller, a set of tracks of a volume in which dataloss has occurred in the tertiary storage controller; suspending, by thetertiary storage controller, another peer to peer remote copy operationbetween the tertiary storage controller and the secondary storagecontroller; and transmitting, by the tertiary storage controller,information on the set of tracks of the volume in which the data losshas occurred to the secondary storage controller.
 27. A systemcomprising, a memory; and a processor coupled to the memory, wherein theprocessor performs operations, the operations comprising: determining,by a secondary storage controller, one or more tracks of one or morevolumes in which data loss has occurred in the secondary storagecontroller; suspending, by the secondary storage controller, a peer topeer remote copy operation between the secondary storage controller anda primary storage controller; and transmitting information on the one ormore tracks of the one or more volumes in which the data loss hasoccurred to the primary storage controller, wherein a timer is queued toprocess a lost track volume list for a predetermined amount of time, andfor each additional track that is lost the timer is requeued once againfor the predetermined amount of time to dispatch a task to process thelost track volume list once no data loss has been observed for at leastthe predetermined amount of time.
 28. The system of claim 27, wherein inresponse to a process volume list task being dispatched once no dataloss has been detected for the predetermined amount of time, cyclingthrough a list and for volumes affected by data loss calling a copyservice code component with a range of peer to peer remote copysecondary tracks.
 29. The system of claim 27, the operations furthercomprising: accumulating over a period of time, by the secondary storagecontroller, the information on the one or more tracks of the one or morevolumes in which the data loss has occurred, prior to transmitting ofany part of the information to the primary storage controller.
 30. Thesystem of claim 27, wherein the one or more tracks comprise a firsttrack and a second track of a volume, the operations further comprising:in response to an unsuspension of the peer to peer remote copyoperation, receiving by the secondary storage controller, data stored inthe primary storage controller to restore data in a range of tracksstarting from the first track to the second track of the volume of thesecondary storage controller to recover from the data loss.
 31. Thesystem of claim 27, wherein a data structure is updated in the primarystorage controller to indicate that data of the one or more tracks ofthe one or more volumes are to be transmitted on continuation of thepeer to peer remote copy operation to the secondary storage controller.32. The system of claim 27, the operations further comprising:determining, by a tertiary storage controller, a set of tracks of avolume in which data loss has occurred in the tertiary storagecontroller; suspending, by the tertiary storage controller, another peerto peer remote copy operation between the tertiary storage controllerand the secondary storage controller; and transmitting, by the tertiarystorage controller, information on the set of tracks of the volume inwhich the data loss has occurred to the secondary storage controller.33. A computer program product, the computer program product comprisinga computer readable storage medium having computer readable program codeembodied therewith, the computer readable program code configured toperform operations, the operations comprising: determining, by asecondary storage controller, one or more tracks of one or more volumesin which data loss has occurred in the secondary storage controller;suspending, by the secondary storage controller, a peer to peer remotecopy operation between the secondary storage controller and a primarystorage controller; and transmitting information on the one or moretracks of the one or more volumes in which the data loss has occurred tothe primary storage controller, wherein a timer is queued to process alost track volume list for a predetermined amount of time, and for eachadditional track that is lost the timer is requeued once again for thepredetermined amount of time to dispatch a task to process the losttrack volume list once no data loss has been observed for at least thepredetermined amount of time.
 34. The computer program product of claim33, wherein in response to a process volume list task being dispatchedonce no data loss has been detected for the predetermined amount oftime, cycling through a list and for volumes affected by data losscalling a copy service code component with a range of peer to peerremote copy secondary tracks.
 35. The computer program product of claim33, the operations further comprising: accumulating over a period oftime, by the secondary storage controller, the information on the one ormore tracks of the one or more volumes in which the data loss hasoccurred, prior to transmitting of any part of the information to theprimary storage controller.
 36. The computer program product of claim33, wherein the one or more tracks comprise a first track and a secondtrack of a volume, the operations further comprising: in response to anunsuspension of the peer to peer remote copy operation, receiving by thesecondary storage controller, data stored in the primary storagecontroller to restore data in a range of tracks starting from the firsttrack to the second track of the volume of the secondary storagecontroller to recover from the data loss.
 37. The computer programproduct of claim 33, wherein a data structure is updated in the primarystorage controller to indicate that data of the one or more tracks ofthe one or more volumes are to be transmitted on continuation of thepeer to peer remote copy operation to the secondary storage controller.38. The computer program product of claim 33, the operations furthercomprising: determining, by a tertiary storage controller, a set oftracks of a volume in which data loss has occurred in the tertiarystorage controller; suspending, by the tertiary storage controller,another peer to peer remote copy operation between the tertiary storagecontroller and the secondary storage controller; and transmitting, bythe tertiary storage controller, information on the set of tracks of thevolume in which the data loss has occurred to the secondary storagecontroller.